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Summary. The paper comments on properties of the so-called "Unified approach to the con- 
struction of classical confidence intervals", in which confidence intervals are computed in a 
Neyman construction using the likelihood ratio as ordering quantity. In particular, two of the 
main results of a paper by Feldman and Cousins (F&C) are discussed. It is shown that in the 
case of central intervals the so-called flip-flopping problem, occuring in the specific scenario 
where the experimenter decides to quote a standard upper limit or a confidence interval de- 
pending on the measurement, is due to an expectation which is not justified. The problem can 
be easily avoided by choosing appropriate confidence levels for the standard upper limits and 
confidence intervals. In the F&C paper "upper limit" is defined as the upper edge of a confi- 
dence interval, whose lower edge coincides with the physical limit. With this definition of upper 
limit (F&C limit), in an approach which uses the likelihood ratio as ordering quantity, two-sided 
confidence intervals automatically change over to "upper limits" as the signal becomes weaker 
(Unified approach). In the present paper it is pointed out that this behaviour is not a special 
property of this approach, because approaches with other ordering principles, like central in- 
tervals, symmetric intervals or highest-probability intervals, exhibit the same behaviour. The 
term "Unified approach" is therefore equally well justified for these approaches. The Unified 
approach is presented in the F&C paper as a solution to the flip-flopping problem. This might 
suggest that the F&C limit is a standard upper limit. Because the F&C limit can be easily mis- 
understood as standard upper limit, its coverage properties are investigated. It is shown that 
the coverage of the F&C limit, if it is interpreted as standard upper limit, depends strongly on 
the parameter y, to be determined, with values around and > (1 + a)/2, where a is the con- 
fidence level of the confidence belt. Differences between the F&C limit and a standard upper 
limit were already pointed out in the F&C paper. In order to exclude any misunderstanding, it 
is proposed in the present paper to call the F&C limit "upper edge of the confidence interval", 
even if its lower edge coincides with the physical limit. 
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1. Introduction 

In 1998 the paper "Un i fied a pproach to the classical statistical analysis of small signals" 



Feldman and Cousins I (|1998l )] appeared, in which the classical approach to the construction 
of confidence intervals is discussed in detail. The paper has received great attention and has 
stimulated the discussion about the calculation of confidence intervals. Two of the main 
results of the paper can be stated as follows: 

• By using the likelihood ratio as ordering quantity one obtains confidence intervals 
which automatically change over from two-sided intervals to upper limits as the signal 
becomes statistically less significant (" Unified approach" ) . 
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• This eliminates undercoverage caused by basing this choice (of quoting an upper limit 
or a confidence interval) on the data (" flip- flopping" ) . 

These attractive features have induced many experimenters to follow the approach pro- 
posed by Feldman and Cousins. The purpose of the present paper is to point out possible 
misunderstandings of the above statements and to give alternative solutions to the flip- 
flopping problem. 



Another result of the paper [Feldman and Cousins I ([1998)] is, that an approach which 
uses of the likelihood ratio as ordering quantity yields finite and physical intervals, in cases 
where central intervals are empty or unphysical. This subject is not discussed here. 



2. Neyman construction of confidence intervals and coverage 



To be specific the same example is considered as in (Feldman and Cousins I (|f 998T )] . a Poisson 



process with background, where the mean background is known: 

(n 4- b) n 

P(„| M ) = [-Gi + &)] (1) 

n is the measured number of signal plus background events 

/i is the average number of signal events 

b is the average number of background events, b is assumed to be exactly known 

P(n\fi) is the probability of measuring n, given /i and b 



For th e discussion in this Section and for later comparisons with the results from [Feldman and Cousins 
(1998)], the likelihood ratio 



„/ x P(n\u) , s 

Rfati = p , | ; (2) 

is assumed as ordering principle, when constructing the acceptance region in n for an 
individual //. fibest is the value of [i which maximizes P(n\fx), at fixed n, where only 
physically allowed values of [i are considered. For a given /x, values of n are added to the 
acceptance region [ni(/i), n 2 (/i)] in decreasing order of the ratio R(n,/j,), until the sum 
p = ^2 P{n\fx) agrees with the desired confidence level a. Such an ordering principle is 
naturally implied by the theory of likelihood ratio tests. 

For a given measurement n, the confidence interval / = /i 2 ] for is found by 
including in / all those fj, which contain n in their acceptance region. The limits /Xi and /i 2 
obviously depend on n. The set of confidence intervals [/ii(n), /x 2 (n)] is called confidence 
belt. 

Upper limits of \x are determined in the same way as confidence intervals, except that 
the ordering quanity is now R(n, fi) — n. The resulting " acceptance regions" are one-sided 
in this case: [ni ow (/J-), oo]. The upper limit of fi at fixed n is given by that for 

which the lower edge niow(p) of the acceptance region coincides with n. In general, upper 
limits of fi at a certain confidence level differ from the upper ends of confidence intervals at 
the same confidence level (see below). 

Considering an ensemble of experiments with arbitrary but fixed /i, this fixed value of 
[i will be contained in the confidence belt in a fraction a of all experiments. This follows 
from the way the confidence intervals are constructed: Given n, the confidence interval 
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is the set of all those /i which contain n in their acceptance region. Confidence intervals 
with this property are said to have exac t coverage . Cov erage is an important feature of 
Neyman-constructed confidence intervals Nevman I (|1937T )]. 

For upper limits the corresponding statement reads: Given an ensemble of experiments 
with arbitrary but fixed fx, this fixed value of /i will be less than /i up (n) in a fraction a 
of all experiments. In the following, this definition of upper limit will be called "standard 
definition" , and the corresponding upper limit as " standard upper limit" . 

It is evident that for both the confidence belt and for the standard upper limit the spec- 
ification of the confidence level is essential. For the confidence belt, the ordering principle 
is relevant in addition. The ordering principle for the standard upper limit is fixed. 

In the present example small under or overcoverage occurs due to the discreteness of 
n. In order to avoid this problem, in the calculations presented below the factorial in the 
discrete Poisson function P(n\fx) is replaced by Euler's Gamma function, which is then 
normalized to 1. Euler's Gamma function, which is defined for all real values of n, agrees 
with the discrete Poisson function at all integer arguments n. The normalization factor 
differs from 1 by less than 0.1% for fi > 3, the maximum difference occuring at fi = 0, 
where it is less than 2 %. 

It should be stressed that it is not proposed to do this replacement in practical appli- 
cations. It is applied here only for the purpose of this paper, to allow a discussion which is 
not affected by under or overcoverage due to the discreteness of n. 

In the general Neyman construction the ordering principle is not specified. In the present 
paper, besides the ordering principle based on the likelihood ratio ([2]) also the ordering 
principle based on central intervals is considered. 



3. Discussion of Neyman-constructed confidence intervals and standard upper lim- 
its 

In the previous Section the construction of confidence intervals and of standard upper limits 
is fully defined. In order to understand that this procedure of defining confidence intervals 
and standard upper limits ensures coverage, it is instructive to consider how coverage would 
be tested by Monte Carlo simulations: One would generate an ensemble of experiments 
with fixed /io, throwing n according to eq.([T]). An acceptance region [ni(/io), "2(^0)] would 
be defined as explained in the previous Section. Although the confidence belt is not yet 
completely defined by knowing the acceptance region for a single /io, one can already say 
that the confidence belt would contain /io in a fraction a of all cases, because for those 
n which lie in the acceptance region of /io (which happens in a fraction a of all cases) /io 
would be added to the confidence interval [/ii(n), fj,2(n)]. These statements are valid for all 
physically allowed values of \i. One can conclude that coverage is confirmed. 

Thus, the way these confidence intervals and standard upper limits arc defined ensures 
exact coverage (disregarding effects which occur when the problem involves discrete values) . 
Obviously, the construction is independent of the decision, whether to quote a standard up- 
per limit, a confidence interval or both. Therefore, also coverage is guaranteed independent 
of such a decision. The experimenter may thus decide to quote a standard upper limit, a 
confidence interval or both, at equal or different confidence levels, without violating cover- 
age. Here, "coverage" is understood as explained in Section [2j Coverage in a very specific 
scenario is discussed in Section S) 

It should also be noted that coverage has to be obeyed at fixed /io, and for coverage it is 
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Table 1. Integrated probabilities p u p and 
P2 below, in an above the acceptance re- 
gions respectively, as functions of p.The 
confidence level chosen is a = 90%. 





Pi 


P 


P2 


0.0 


0.000 


0.9 


0.100 


0.5 


0.000 


0.9 


0.100 


1.0 


0.016 


0.9 


0.084 


1.5 


0.035 


0.9 


0.065 


2.0 


0.046 


0.9 


0.054 


2.5 


0.053 


0.9 


0.047 


3.0 


0.056 


0.9 


0.044 


3.5 


0.057 


0.9 


0.043 


4.0 


0.057 


0.9 


0.042 


4.5 


0.057 


0.9 


0.043 


5.0 


0.056 


0.9 


0.044 


5.5 


0.056 


0.9 


0.044 


6.0 


0.056 


0.9 


0.044 


6.5 


0.056 


0.9 


0.044 


7.0 


0.056 


0.9 


0.044 


7.5 


0.056 


0.9 


0.044 



Table 2. Lower and upper edge (ni and n 2 ) of the 90% 
c.l. acceptance region of n, 90% c.l. lower limit and 95% 
c.l. lower limit of n, as functions of p. 





m 


n 2 


"90% (P) 


"95%(^) 


0.0 


0.00 


5.34 


1.00 


0.62 


0.5 


0.00 


6.01 


1.28 


0.84 


1.0 


0.53 


6.91 


1.59 


1.09 


1.5 


1.13 


7.92 


1.92 


1.36 


2.0 


1.59 


8.85 


2.26 


1.65 


2.5 


2.01 


9.70 


2.61 


1.96 


3.0 


2.39 


10.48 


2.98 


2.28 


3.5 


2.75 


11.20 


3.35 


2.61 


4.0 


3.09 


11.87 


3.72 


2.95 


4.5 


3.43 


12.51 


4.10 


3.29 


5.0 


3.78 


13.15 


4.49 


3.65 


5.5 


4.14 


13.79 


4.87 


4.00 


6.0 


4.50 


14.43 


5.26 


4.36 


6.5 


4.87 


15.06 


5.66 


4.73 


7.0 


5.23 


15.69 


6.05 


5.09 


7.5 


5.61 


16.32 


6.45 


5.47 
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Table 3. Lower and upper edge (in and ^2) of the 90% 
c.l. confidence interval of ii, 90% c.l. standard upper 
limit and 95% c.l. standard upper limit of fi, as functions 
of n. 



n 


Mi 


M2 


M90% (n) 


M95%(") 


0.0 


0.00 


0.61 


0.00 


0.00 


0.5 


0.00 


0.98 


0.00 


0.00 


1.0 


0.00 


1.38 


0.02 


0.86 


1.5 


0.00 


1.90 


0.87 


1.75 


2.0 


0.00 


2.50 


1.64 


2.58 


2.5 


0.00 


3.16 


2.36 


3.35 


3.0 


0.00 


3.87 


3.04 


4.08 


3.5 


0.00 


4.59 


3.71 


4.81 


4.0 


0.00 


5.30 


4.39 


5.50 


4.5 


0.00 


6.01 


5.04 


6.21 


5.0 


0.00 


6.69 


5.67 


6.89 


5.5 


0.11 


7.36 


6.31 


7.56 


6.0 


0.49 


8.04 


6.94 


8.21 


6.5 


0.81 


8.69 


7.56 


8.89 


7.0 


1.04 


9.34 


8.19 


9.54 


7.5 


1.29 


10.01 


8.81 


10.19 


8.0 


1.54 


10.64 


9.41 


10.84 


8.5 


1.81 


11.29 


10.03 


11.47 


9.0 


2.08 


11.93 


10.64 


12.11 


9.5 


2.37 


12.56 


11.24 


12.74 
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Fig. 1. 90% c.l. confidence belt (full circles), 90% c.l. standard upper limit (stars) and 95% c.l. 
standard upper limit (open circles) for the average number of signal events p in the presence of a 
Poisson background with known mean b = 3.0. 



irrelevant how far /iq is away from the edges /ii(n) and /J. 2 (n) or whether fJ.i(n) is given by 
the lowest physically allowed value of /i. This is characteristic for a Frequentist approach. 
The discussion in Section [5] will refer to this feature. 

Assuming a confidence level of a — 90%, setting b equal to 3 and using the likelihood 
ratio as ordering quantity, one obtains for the present example the confidence belt shown 
in FigQ](full circles). Also shown in FigQ]are the 90% c.l. standard upper limit (stars) and 
the 95% c.l. standard upper limit (open circles) of /x. 

As can be seen, the 90% c.l. standard upper limit and the 90% c.l. confidence belt in 
Fig[T]are perfectly c ompatible with the correspond ing limit and confidence belt respectively 



in Figs. 5 and 7 of [Feldman and Cousins I ([1998)] . Replacing the discrete Poisson distri- 
bution by the renormalized Euler's function is therefore a valid procedure, for the purpose 
of this paper. 

Table [1] gives for each n the integrated probabilities below, in and above the acceptance 
region of n: Pl = J n<nl P(n\fi)dn, p = /„ x < n < n2 P(n\fi))dn, p 2 = J n>n2 P{n\n))dn. By 
construction, p is always equal to a = 90%. As a result of the ordering principle chosen, 
Pi is generally different from p 2 , p\ being less than p 2 for /i < 2.4 and greater than p 2 for 
At > 2.4. 

In Table [5] the lower and upper edge (n\ and n 2 ) of the 90% c.l. acceptance region of 
n, the 90% c.l. lower limit n 90 o/ (p) of n and the 95% c.l. lower limit n^%(p) of n are 
given as functions of \i. The inverse functions of ni(fi), n 2 (p), n 90 %(/j,) and ng 5 %(/i) are 
the functions n 2 (n), Hi(n), /igo%(^) and ^95% (n) respectively, which define the confidence 
intervals and standard upper limits of /i as functions of n. They are listed in Table [3J 

By definition, for a fixed fio, n lies within the acceptance region [rii(/io), "2(^0)] if and 
only if /iQ lies in the confidence belt [/Lti(n), fi 2 (n)]. This means that the probability that 
[/ii(n), [i 2 (n)] covers the value /x is equal to p = a. 

Also by construction, n > ni(p) if and only if /x < [i 2 (n), implying that the probability 
for [i < ^ 2 (n) is equal to p + p 2 . Thus [/ii(n), fJ, 2 (n)} is a confidence interval at the a- 
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confidence level. If ^{n) is interpreted as a standard upper limit, its coverage (p + P2) 
depends strongly on fj,, making this interpretation problematic. The situation is different 
for central intervals, for which pi = (1 — a)/2, p = a and p 2 = (1 — a)/2. In the case of 
central intervals, ^2 is a standard upper limit at the confidence level p + P2 = (1 + a)/2, 
at all ijl. 



4. The flip-flopping problem 

In this Section a specific scenario is considered, in which the experimenter decides to quote 
a standard upper limit at the c.l. if n is below and a confidence interval at the c.l. a if n is 
above a certain value no- It should be noted that this very specific scenario is different from 
the scenario which is assumed when constructing confidence intervals or standard upper 
limits (see Section [5J . In the latter scenario one talks independently about coverage for 
a Neyman-constructed confidence interval or coverage for a Neyman-constructed standard 
upper limit, and there is no condition involving the measurement n. Therefore, the coverages 
for the two scenarios are not expected to be identical. In addition, the coverage 7 for the 
specific scenario depends on the choice of a and 0. For central intervals all coverages of 
interest can be directly given, without the need of complicated calculations. 



In [Feldman and Cousins I ([1993)] the special case a — = 90 % is discussed for an 



approach with central intervals. One obtains aco yerageof 7 = 0— P2 = a—p2 = (3a— 1)/2 = 



85 %, at low p,. The fact that 7 ^ a is called in [Feldman and Cousins I (|1998f )] violation of 
coverage, or flip-flopping problem. Talking of "violation of coverage" means that a coverage 
of 7 = a is expected. As pointed out above, there is no justification for that. 

The actual problem is that 7 is different for different regions of p. If 7 were the same 
for all fi, one could quote a unique confidence level (= 7) for the specific scenario and there 
would be no problem. 

The behaviour 7 ^ a at low p is due to the fact that with the chosen alternative between 
a standard upper limit (at a c.l. 0=90 %) and a confidence interval (at the c.l. a=90%) 
the upper edge ^{n) of the confidence belt, in the region n < hq, is shifted. Knowing this, 
it is easy to find a way of avoiding the flip-flopping problem: One has to define alternatives 
which preserve the upper edge (12(11), at all fi. In the case of central intervals the alternatives 
are a (1 + a)/2 = 95% c.l. standard upper limit and a a = 90% c.l. confidence interval. 
With this choice the upper edge p2(n) of the confidence belt is not changed, because for 
central intervals ^(n), if interpreted as standard upper limit, has a c.l. of 95%. Since the 
upper edge of the confidence belt is preserved, coverage is fulfilled. Thus the flip-flopping 
problem can be easily avoided by choosing appropriate confidence levels for the standard 
upper limit and the confidence intervals. 



5. The approach by Feldman & Cousins 

In the approach described in Feldman and Cousins (1998)] a confidence belt is determined 
according to a Neyman construction, where the likelihood ratio is used as ordering quan- 
tity. Coverage is therefore guaranteed for the confidence level a chosen: If an experiment 
with fixed po were repeated many times one would obtain a set of confidence intervals 
[pi(n), H2(n)}i and the relation pi(n) < po < p>2(n) would be correct in a fraction a of all 
cases. Because for small n the lower edge Hi(n) coincides with the physical limit of /x, the 
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set of relations (which define the confidence belt) can also be written as 

Mo < / i 2(") f° r n < n (a) 

(3) 

< fi < fJ-2(n) for n > n (b) 

where uq is the largest n for w hich ni(n) coincides with the physical limit. For n < uq, 



fj-2{n) is called "upper limit" in [Feldman and Cousins I ([1998)], and it will be called F&C 



limit in the following: the F&C limit is defined as the upper edge of a confidence interval, 
whose lower edge coincides with the physical limit of fi. Again, out of all relations ([3|) a 
fraction a of them would be staisfied, and coverage would be fulfilled. However, one should 
not call the F&C limit an "a c.l. upper limit", because the relation < ^(n) is not 
satisfi ed in a fraction a of all cases, and, moreover, this fraction depends on 

In Feldman and Cousins I ( 19981 )] the presence of two kinds of intervals ((a) and (b)) is 



expressed as " This choice (of using the likelihood ratio as ordering quantity) yields intervals 
which automatically change over from upper limits to two-sided intervals as the signal 
becomes more statistically significant (Unified approach)". 

This is correct, when the term "upper limit" is understood as the F&C limit. One should 
note, however, that this is not a special feature of an approach which uses the likelihood 
ratio as ordering quantity. Approaches with other ordering principles, like central intervals, 
symmetric intervals or highest-probability intervals, exhibit the same behaviour. The term 
"Unified approach" i s therefore equally well j ustifie d for these approaches. 

The approach in Feldman and Cousins I ( 19981 )] is also presented as a solution to the 



flip-flopping problem (see Sec tion [4]) . This could be understood such that the approach in 
Feldman and Cousins 1 1998f )] provides an alternative between a standard upper limit and 



a confidence interval. However, this approach rather provides an alternative between the 
F&C limit and a confidence interval. The F&C limit can therefore be easily misunderstood 
as a standard upper limit. 

As noted in Section [3j in a Frequentist approach the value of the physical limit of fi is 
irrelevant, except that the acceptance regions for n can only be determined for physically 
allowed values of \i. The definition of "upper limit" as the upper edge of a confidence inter- 
val whose lower edge coincides with the physical limit, is therefore not a Frequentist type of 
definition and may lead to confusion. In order to exclude any misunderstanding it is recom- 
mended in the present paper to call the F&C limit "upper edge of the confidence belt", even 
if the lower edge coincides with the physical limit of fi. Differences between the FfcC limit 



and th e standard upper limit were also pointed out by the authors of [Feldman and Cousins 
(I1998J) ] 



From this discussion it follows, that the coverage properties of the F&C limit, if in- 
terpreted as a standard upper limit, are of great interest. From Table [1] one can see that 
(for a — 90%) this coverage (p + P2) varies between 94,4 % and 100 %, Compared to 
the confidence level a of the confidence belt, the F&C limit ^2, if interpreted as standard 
upper limit, is very conservative, with a coverage around (l + a)/2=95% or an overcoverage 
around (1 + a)/2 - a = (1 - a)/2=5%. 

In the case of central intervals the upper edge of the confidence interval has the nice 
property that it is a standard upper limit with a well-defined confidence level, namely 
(1 + a)/2, at all fi. 

Coming back to the flip-flopping problem: In the approach by F&C this problem is 
avoided, because introducing the alternative between an F&C limit and a confidence interval 



Comments on the Unified approach ... 9 



doesn't change the original confidence belt, for which coverage is fulfilled. This is analogous 
to the approach with central intervals, where the original confidence belt is preserved by 
choosing the alternative between a (l+a)/2 c.l. standard upper limit and a a c.l. confidence 
interval (see Section 2]). 



6. Summary and Conclusions 



The paper discusses two of the main results of [Feldman and Cousins I (119981)1. One of them 



concer ns the so-called flip-flopping problem, which according to [Feldman and Cousins 



( 19981 )] exists for central intervals in the specific scenario, where the experimenter decides 



to quote a standard upper limit or a confidence interval, depending on the observed data. 
In the present paper it is shown that this problem is due to an expectation which is not 
justified. The problem can be easily avoided by choosing appropriate confidence levels for 
the standard upper li mit and the confidence belt. 
Another result of 



Feldman and Cousins 



Feldman and Cousins I ([19981 )] concerns the Unified approach. In 
( 19981 )] the term "upper limit" denotes the upper edge of a confi- 



dence interval whose lower edge coincides with the physical limit of ji. With this definition 
of upper limit, upper ends of confidence intervals automatically change over to "upper lim- 
its" as the signal becomes weaker. This transition from confidence intervals to upper limits 
is not a special property of the approach using the likelihood ratio as ordering quantity, 
because the same transition takes place in approaches with different ordering priciples, like 
central intervals, symmetric intervals or highest-probability intervals. Thus the term "Uni- 
fied approach" is equally well justified for these approaches. The approach with central 
intervals has the additional advantage that the upper end of the confidence interval is a 
standard upper limit at a well defined confidence level, at all fj,. 

The Unified approach is presented in the F&C paper as a solution to the flip-flopping 
problem, where the experimenter quotes alternatively a standard upper limit or a confidence 
interval, depending on the measurement. The F&C limit can therefore easily be misunder- 
stood as a standard upper limit. For this reason the coverage properties of the F&C limit 
are investigated. It is shown that the coverage of the F&C limit, if it is interpreted as 
standard upper limit, depends strongly on the parameter /j, to be determined. Compared 
to the confidence level a of the confidence belt, the upper limit is very conservative, with 
a coverage around (1 + a)/2 or an overcoverage around (1 + a)/2 — a = (1 — a)/2. More 
precisely, for a = 90 %, the coverage of the upper limit varies between 94,4 % and 100 %, 
and a unique confidence level cannot be assigned to it. Dif ferences between the FfcC lim it 
and a standard upper limit were already pointed out in Feldman and Cousins ( 1998f )]. 



In order to exclude any misunderstanding, it is proposed in the present paper to call the 
F&C limit " upper edge of the confidence interval" , even if its lower edge coincides with the 
physical limit. 

In this paper only one example was discussed in detail. The conclusions and recommen- 
dations, however, are also valid for any other application, in which confidence and upper 
limits are determined using the likelihood ratio as ordering quantity. Though, the appli- 
cations are restricted to the cases with only one unknown (/i) and no nuisance paramter, 
except if the nuisance parameter is exactly known (6). 

Finally it should be emphasized that the criticism expressed in this paper does not refer 
to the Neyman construction of confidence intervals itself or to the use of the likelihood ratio 
as ordering quantity. 
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